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BACKGROUND AND PHILOSOPHY 

The CSE expository and narrative rating scales have been developed to 
meet the need for sound, instructional^ relevant methods for assessing 
students 1 writing competence. Each scale's analytic rating elements are 
referenced to " specific rhetorical features characteristic of the writing 
aim addressed — exposition or narration. This referencing is based on our 
belief that students 1 writing in each mode can be analyzed into its consti- 
tuent elements. Knowledge of students 1 performance with respect to these 
elements can provide valuable information for assessing achievement and 
facilitating instructional planning at the classroom, school, and district 
levels. 

The CSE analytic scales reflect our view of the roles of evaluation 
and testing in instruction. We are convinced that testing and instruction 
can become complementary activities and that both need to share the same 
educational goals and objectives. In the case of writing, for example, 
criteria used to assess students 1 compositions ought to reflect agreed-upon 
standards of good writing. Classroom instruction should address explicitly 
these same standards. In this kind of system, assessment can play a valu- 
able role in instructional improvement by identifying specific areas of 
strength and weakness for individual students, and by indicating areas 
where classroom, school, and district curricula may need to be improved. 

The CSE scales employ analytic rating procedures to meet the demands 
of the kind of assessment system described above. Raters assign points to 
each of several aspects of a student's composition, providing a rating of 
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the overall quality of the written product as well as ratings on specific 
elements. In contrast to the undifferentiated score provided by holistic 
scor i ng ' procedures ,. the CSE :>cal es 1 overal 1 rati ng and rati ngs on the 
specific elements give concrete 'information to guide instructional 
planning. 

To be sure, some c<psts in rater time are involved in scoring written 
work analytically. Our studies indicate that it takes raters four to. five 
minutes to assign analytic ratings to a multi-paragraph writing sample, and 
two to four minutes for a single paragraph. In comparison, it may take 
only about one minute to assign a holistic rating to a fairly short writing 
sample. 

While we believe that the greater time commitment required by analytic 
rating is offset by instructional advantages, economics may require some 
compromi ses . For exampl e , where student - mastery and money are seri ous 
issues, the scale can be used to provide analytic, ratings only for those 
students falling below mastery. In a similar vein, the scale can be abbre- 
viated to target on specific skills of relevance to classroom instruction 
and ongoing classroom assessment. 

In the remainder of this paper, we will provide an overview to the 
scales and describe the theory and research forming their base. We will 
describe the features, purposes, and uses of the expository and the narra- 
tive scales, and the training that CSE provides for school systems wishing 
to use the scales in their writing assessment programs. 



OVERVIEW TO THE SCALES 



We mentioned earlier that the CSE scales reflect the belief that 
students 1 writing can be analyzed into its constituent elements. We 

pointed out that if we can get accurate information about- students 1 perfor- 

/ 

mance on these writing elements we can use it for instructional planning 
and improvement^ 

In the fol lowing sections we describe the theoretical, and research 
basis for our belief that students' writing can be structurally analyzed. 
The work we describe led to the development of the analytic elements in our 
scales. 

The Theoretical and Research Base 

Writing is a very complex skill. While most people would probably 
agree with that statement, some people may assume that "a good writer is a 
good writer" and that a student who writes well in one mode will write 
equally well in another. The research points out some dangers in that 
assumption. 

We can ask students to write in several modes of ^tscourse or dis- 
course aims. The two modes most commonly found in school curricula are 
expository and narrative writing. Expository writing involves expressing 
facts and ideas, and usually requires the student to support the ideas 
presented with appropriate detail, explanation, and logical development of 
thought. Narrative writing rel ates experiences and events, and often 
requires the student to provide chronological development and description 
of events. 
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We believe that a student called on to write jn the expository mode 
will need to draw on different skills, reflecting different discourse 
structures, than he or she would need to achieve a narrative aim. For 
example, if a student is to explain something in the expository mode, he or 
she would need to present the main idea, develop the idea with supporting 
details, and perhaps summarize the issue presented. On the other hand, if 
a student is asked to give an account of something in the narrative mode, 
he or she might need to develop a sequence of events and describe the 
setting and characteristics with sufficient detail to make the account 
believable. 

The research findings support our view of these differing skill 
needs. There is a good deal of evidence from studies of students 1 reading 
abil ity that different skills are needed to understand text written in 
different modes. There is similar evidence from studies of students 1 
writing ability that different skills are needed to produce written text in 
different modes. 

For instance, a student reading a passage written in the expository 
mode relies on a different set of organizing schemes, and different methods . 
of breaking down, classifying, and understanding what is written than he on 

; _ . . . . ' ..." - - ..../> 

she would draw on to read a passage in the narrative mode (Meyer, 1975; 
Graesser, Hauft-Smith, Cohen, & Pyles, 1979). 

Just as students use different processes to read and make sense of 
different kinds of writing, they also use different skills when they are 
asked to write in different modes such as exposition and narration (Veal & 
Tillman, 1971; Praeter & padia, 1980; Quellmalz, Capell, & Chou, 1982.) 
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We mentioned earlier that w& are interested in tying together testing 
and instruction. For other people sharing this interest the research 
offers a clear message: different kinds of writing assignments require 
different kinds of student writing strategies and skills. 

But there is a problem here. In many schools 1 writing programs, 
students are normally asked to compose in only one mode ~ expository or 
narrative. Now, a student might have greater ability in writing a narra- 
tive assignment stressing chronological development than he or she has in 
writing an expository assignment requiring logical development. The oppo- 
site could also be true. 

A teacher might get one kind of picture, therefore, about a student's 
writing ability based on how he or she develops narrative. The teacher 
might get a different picture about a student's writing ability based on 
how he or she develops an expository piece of work. If only one kind of 
writing ability is sampled, then it is possible that some writing defi- 
ciencies will go undiagnosed. 

The point we are making here is simple: we cannot accept the assump- 
tion that a "good writer is a good writer." It depends on what we askythe 
student to write and for this reason we developed scales for the two dis- 
course aims most frequently used — expository and narrative. 

The CSE scales are important, therefore, because they provide informa- 
tion on students 1 expository and narrative abilities. They are equally 
important because of the way they assess these abilities. 

There are two primary, ways of assessing students 1 writing performance 
— constructed response and selected response. A constructed reponse 



provides a direct measure of a student's writing ability. Here, the 
student is asked to write sc lething, perhaps a paragraph or two, perhaps an 
essay. When the piece of writing is turned in, a teacher or some other 
person judges the quality of the student's writing. 

A selected response provides an indirect measure of a student's 
writing ability. Frequently, the student is given a passage to read 
followed by a multiple-choice test about the passage. ^Thls^test^m^-as^ 
questions about the organization of the written passage, its supporting 
evidence, its grammar and mechanics. The score the student gets on the 
test is supposed to show how much the student knows about writing. And so 
far, maybe, so good. But there is a problem if we % l;e the inference that 
student knowledge of writing quality accurately indicates student ability 
to produce good writing. If this inference is unjustified, then once again 
students 1 writing deficiencies may remain undiagnosed. | 

Several reasons are offered for using indirect, multiple-choice tests 
of students 1 writing ability. First, multiple-choice tests* take less time 
to score than essays do. Second, some people believe that there is a 
strong relationship between students 1 multiple-choice test scores and 
their written work. Third, multiple-choice tests are more objective than 
ratings of students 1 written work. The argument is that if two people 
score the same essay in some general, impressionistic way, there is a good 
chance they will arrive at different judgments about the same piece of 
writing. Let us take up these three arguments. 

Argument one: Scoring essays does take more time than scoring a mul- 
tiple-choice test. However, our scales have been designed so that they do 



not take up exorbitant amounts of rater scoring time. Depending on the 
length of the writing sample and rater familiarity with the scale, rating 
time ranges from two to five minutes. We believe the diagnostic and 
prescriptive information returned justifies the time invested. 

'if l 

Argument two: There is no guarantee that a student who scores well 
on a multiple-choice test of writing will be able to produce good writing. 
Researchers worki ng speci f ical ly i n wri ting do not be! i eve that such 
indirect measures as multiple-choice tests provide an accurate indication 
of students 1 writing ability (Braddock, et al., 1963; Cooper & Odell, 
1977; Quellmalz, Cappell, & Chou, 1982). If these tests are well 
constructed (and that is always a big if no matter what their purpose is) 
then they may be fair measures of reading comprehension , but not of how 
well a student will produce a piece of writing . 

Argument three: The CSE scales have built-in procedures to make sure 
that different judges of a student's writing use the same decision rules. 
There is less liklihood, therefore, that they will give greatly different 
scores or grades to the same piece of work. 

At CSE we built on some of the research mentioned above in our own 
work on the different writing and response modes (Quellmalz, Capell, & 
Chou, 1982). In our studies, we looked carefully at whether students 1 
writing abilities in the two major modes -- exposition and narration — 
were comparabl e. We al so examined whether different response modes — 
written work versus a multiple-choice test -- provided the same kind of 
information about student writing ability.. 

Her* is a quick sketch of what we found: 

° Students 1 writing skills did differ in the two discourse modes. 



° While ratings of exposition were generally different than those 
of narration, the mechanics scores on the two modes were 
comparable. Perhaps this helps explain the false notion 
that "good writing is good writing." 

° Multiple-choice scores were poorer indicators of student 
writing ability than > measures based on actual student 
writing samples. 

After these spx&res were Conducted and the information analyzed, the 
scales we developed for the study were refined and now take the form we 
describe below. \. 
The Expository Scale 

This scale is used to assess how well a student can write in the 
expository mode. By exposition, we have in mind writing that .intends to 
inform by presenting facts, giving directions, recording events, inter- 
preting facts, developing opinions. Kow well the student handles these 
elements influences the effectiveness of his or her expository composition. 

The scale uses five elements to assess students' writing-in the expo- 
sitory mode. These are: General Competence; Essay Organization and Coher- 
ence; Paragraph . Coherence; Support; and Grammar /Mechani cs . The first 
subscale, General Competence, is used for making a holistic evaluation of 
the assay's command, of basic writing elements. The four other subscales 
focus on analytic evaluations of the quality of writing. 

Using the General Competence subscale, raters read the composition to 
form a global judgement of how well the student arranged fundamental 
elements. In rating the composition for its general quality, raters keep 



1n mind the question of whether the work achieves an expository purpose for 
the Intended audience. General Competence asks the rater to keep the 
question of fundamental arrangement in mind and give an overall rating 
showing how well the student handles the skills Incorporated 1n the four 
remaining subscales. The rater can assign the composition a global score 
ranging from six to one. Scores of four, five, and six represent verylng 



degrees of mastery; scores of one, two, and three represent varying degrees 
of non-mastery. This s1x-po1nt system, which 1s described 1n greater 
detail later 1n the paper, applies to all the subscales. 

In the remaining subscales, the rater's attention 1s directed to 
specific skills and qualities. Each skill, or quality 1s described 1n 
detail. 

° The Essay Organization and Coherence subscale focuses on the flow of 
Ideas throughout the entire composition and between Its paragraphs. 
Emphasis here 1s on vertical relationships throughout the essay. 
The rater 1s reading to see 1f the 'essay has a main Idea, for 
example, which makes a point about the topic, whether that point 1s 
at a greater level of generality than the other points 1n the paper, 
and 1f the points made relate to the essay's thesis. 

° The Paragraph Organization subscale 1s concerned with horizontal 
relationship within paragraphs, with the logical arrangement of 
points and their subordination to the paragraph topic. 

° The Support subscale focuses on the specificity, depth, and amount 
of elaboration used to develop the theme. 

° The Grammar/Mechanics subscale focuses on errors 1n sentence struc- 
ture or mechanics and how seriously thay Interfere with communica- 
tion. It pinpoints global errors making 1t difficult to understand 
the writer s message, and more local errors which are not serious 
Impediments to communication. 



The Narrative Scale 



This scale 1s used for assessing how well a student writes 1n the 
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narrative mode. By narration* we have in mind a description of an 
event(s), based on an account of such aspects as (l) how, when, and where 
the event(s) took place; (2) the persons, places, or things involved in the 
evtnt(s); and (3) the actions, thoughts, or feelings of the actors 
described. How well the student handles these features influences the 
effectiveness of his or her narrative essay. 

The scale uses four elements to assess narrative writing. These are: 
— General Competence; Focus and Organization; Support; and Grammar/ 
Mechanics. The first subscale, General Competence, is used to make an 
overall or holistic evaluation of the essay's command of writing fundamen- 
tals. The three other subscales are used to make analytic evaluations of 
the specific component features of narrative writing. Again, these three 
subscales focus on specific elements which are described in detail. 

The General Competence is based on a global judgement of the compo- 
sition. The central question is the narrative effectiveness of the work 
and how well it arranges the features incorporated in the remaining sub- 
scales. A six-point scale, with the same values as those described for the 
expository scale, is used. 

° The Focus/Organization subscale is concerned with the composition's 
structure and flow of ideas— within- and between- paragraphs. 

° Support, which has a similar emphasis to that described for exposi- 
tion, focuses the rater's attention on the development of events, 
descri pti ons , and characters through the use of wel 1 -i ntegrated 
details. s 

° The Grammar/Mechanics subscale focuses on the kinds of global and 
local errors in sentence structure and mechanics described above. 
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Features Common to Both Scales 

| The research basis : Each scale, and its specific analytic elements, 

is I firmly anchored in our own and others 1 research on writing and its 

i 

I 

assessment. The global or holistic judgment offered in each scale assumes 
that the whole is greater than the sum of its parts, and asks for a rating 
of the quality with which the writer engages the topic to achieve the \ 
intended purpose. The discrete, analytic features in each scale -ask for 
separate ratings of a fundamental set of elements. . These elements are 
specified because they recur in tbe research as basic features of acceptable 
writing. They also represent priorities in many writing competency 
programs. The individual subscales therefore present specific criteria for 
judging the student's skill in using these elements. 

Setting the scope and focus of the assignment : Each scale is intended 
to be used in an assessment setting in which students are provided with 
clear writing prompts containing explicit directions to help them plan and 
develop their assignment. These directions should include: the specific 
purpose or function of the writing; a specific audience to whom the student 
will write; a specific topic or subject to write about; and criteria" that 
will be used to judge the essay. 

These directions are intended to provide students with a clear and 
common understanding of the task expected of them; that is, the rhetorical 
context. As such, they are the initial step in ensuring that each student 1 s 
writing is judged on the same task description. They are a counterpart to 
the common scale criteria used by raters. 

The six-point rating scale : Each numerical rating on the scale is tied 
to specific criteria and examples of skill levels and deficiency levels. 
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These numerical ratings are meaningful and consistent Indicators of 
students' writing performance because they provide raters with specific, 
guidance to decide upon the numerical 'rating that the composition should 
receive with respect to a given scale. 

On a given- subscale a student receives 'a score Indicating his or her 
degree of mastery of the skills specified for that scale. In exposition, 
for example, for a student to receive a rating of six (highest mastery 
level) on overall organization and coherence, the rater 1s directed to look 
for certain features in the composition. Among these features are: 
limitation of the topic; logical. essay plan; maintenance of essay plan. 

For the student to receive a score of four (lowest level of mastery) 
on this part of the expository scale, among the features the rater 1s 
directed to look for are: clear main Idea, logical plan, and reasonable 

support. ! 

Each of the other subscales used for expository rating offers similar 

criterion levels and -examples. , 

1 

The same level of specificity of guidance to the rater 1s offered 1n 
the narrative scale. For example, oh the support subscale, for a student's 
composition to receive a rating of five (second-highest mastery level) his 
or her work must provide supporting details, such as x examples and. descrip- 
tions, to develop events or characters. A student whose use of detail 1s 
not Integrated would receive a score of two. ». 

Adaptability to local curriculum needs : The subscales, with their 
accompanying operational criteria, should represent Instructional priori- 
ties of the school system using the scales. In an ideally integrated 
assessment and instruction system,, criteria used to evaluate writing 

16 
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gathered "for formal assessment should be the same as those used to evaluate 

and provide feedback to students on their more routine classroom writing 

\ 

ass1gnments.\ While Individual classrooms might focus on additional ele- 
ments such as\ originality of content and style or voice, criteria used 1n 
judging formal^ functional writing should reflect desired core writing 
elements. The CSE scales, therefore, Include these commonly valued basic 
writing elements. 

These core elements are intended to provide starting points for 
schools and teachers to consider as they plan their own writing assessment 
needs, for example, elaborated mechanics elements of sentence construc- 
tion, usage, spelling, punctuation, and capitalization can provide useful 
classroom-level diagnostic information, ^especially for younger or less pro- 
ficient students whose writing shows various mechanical weaknesses. On the 
other hand, for large-scale competency testing programs assessing older or 
more able students, a comprehensive mechanics rating may be sufficient. 

As we will see in^jie^next section, when CSE provides schools and 
districts with training 1n the use of the scales, part of that training 
Involves making sure the criterion descriptions are agreed upon by the 
potential raters. 

Copies of the complete, expanded scales and scoring criteria appear 
1n Appendix A. This expanded version serves primarily as a "text" which 
raters read before trai n1 ng to f ami 1-1 ari re— t-hemse-l-ves— w4-th — the — s cale s 1 
rationale, structure, criteria, and criterion examples. During training 
and actual rating, raters use a shorter, tabular version of the scales. 
Copies of actual student writing samples scored using our scales appear in 
Appendix B. / ■ j 
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TRAINING AND PROCEDURES 
TO ENSURE RELIABILITY 



In the preceding secion we mentioned that students need clear direc- 
tions to make sure that they all bring the same understanding of the 
writing task to the job at hand* We also pointed out that raters need to 
follow the same criteria as they assess students 1 written work. 

This section talks more about the procedures which help ensure that 
raters do Indeed apply the same criteria. It also offers some reasons for 
why raters, even when they do try to use the same criteria, can "drift" 
apart as they go about the job of scoring students 1 written work. The 
training we offer helps overcome this problem. 
Overview to the Problems and their Solutions N 

. Samples of students 1 written work can provide direct evidence of 
writing ability. Scoring these samples, however, can present a problem, 
even with exploit criteria to follow; no matter how good the scoring 
systems, assuring rel 1abil 1ty ' or agreement among raters requires careful 
planning. First, raters who come to a training session with differing 
views of evaluation must accept and then learn how to apply the rating 
scales accurately and consistently. Second, raters who have achieved high 
levels of agreement during training tend to drift apart and show less 
agreement /hen they must score large numbers of papers. This drift may be 
—cau^sed-by-fatlgue-leadi^g-to c aratess-app-Uca-tto n of cri teci a, orJ-t_majL_he^ 
i the result of other Influences, such as the range of quality 1n the papers 
! being scored or the re-emergence of idiosyncratic rater values. 
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What this means, then, is that even when a careful rating system is 
used, the more papers a rater reads the more likely he or she will drift 
away from the intended criteria. The training we provide, which grew out 
of previous CSE research and technical assistance in writing assessment 
(Quellmalz, 1980), has built-in procedures to control this potential drift 
and to keep raters on track. 
Training Sequence 

The training sequence has three basic features. First, it emphasizes 
developing consensus in defining scale elements that are sensitive to local 
needs. For example, the expository scale may present separate subscales 
for focus and organization rather than a combined coherence scale. Also, a 
district can decide to add features of concern in more advanced writing 
such as style or attention to audience. 

Second, specific criteria are provided for each element, and raters 
receive repeated practice and feedback in their application. Third, the 
training serves as precursor to actual ratings of students 1 written tests; 
it is not merely an academic exercise. Therefore, the rating procedures 

are careful ly structured to ensure that raters learn to apply criteria 

j 

accurately and continue to , do so as they go on to rate actual test essays 
independently. A key feature of this /research-based training sequence is 
the; use of essays which have been prescored by a panel; of experts and which 
are then used as examples during training, as qualifying papers at the end 
of training, and as inserted "check" papers during independent scoring to 
identify drifting raters and to help them stay on track. 
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Training proceeds as follows: 



Step 1: 



Step 2: 



Step 3: 



Setting up a Rater Training Session 

CSE and the school or district asking for the 
trai ni ng di scuss overal 1 purposes. Questi ons may 
include: Is training to be provided for one scale 
or both? Will the; scales be used for routine 
cl assroom assi gnments , school /di stri ct competency 
or both? 

The district collects representative student writing 
samples . These may come from pilot tests of prompts 
1n neighboring districts. The samples must be 
available well in advance (three weeks) of the 
scheduled training session. 

The samples should . reflect the spectrum of student 
writing abilities and represent the grade levels 
tested and/or 1n which the raters teach. 



The sample/papers are used for three purposes: 

1. as practice training papers (approximately 30) 

2. as pilot %est papers to be sure that raters 
qualify at the end of training before proceeding 

-to independent rating (approximately. 20 to 30) 

3. as check papers that will be Inserted among the 
essays that will actually be judged during 
independent rating (a set of three papers to- be 
read after every one or two hours of scoring) 

I > 

Using the CSE scale, CSE and district staff prescore 
the samples for use as training, qualifying, and 
check papers. "Solid" and marginal examples 1 rated 
from 1 to 6 on each of the subscales are prepared. 



for the score that: cite features of the papers arid 
use language from the scale' are prepared for all 
check papers and j for about 10 of the training 
papers. / 



"Feedback 




20 
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Step 4: Packets, of the students 1 essays are prepared for 
each rater. The training packet contains cppies of 
the expanded, prose version and the shorter, tabular 
version of . the rating scales, and the training 
essays.. To acquaint raters with the check procedure 
they will encounter later, written feedback . sheets 
are prepared for about 10 of the training papers; 
trainers, however, have notes to explain all the 
prescored trai ni ng papers . Feedback, sheets wi 11 
only be distributed after raters have scored a 
paper. 

The qualifying packet consists of 10 to 20 prescored 
essays that represent the full range of student 
compositions. / '' 

The check\papers may be put in separate packets for 
each rater \ or be available at a central location. 

Test folders/ are prepared containing 10 essays which 
will be rated independently. The 10 papers in each 
folder are randomly selected from the full set of 
student essays to be rated. Each folder is 
numbered. 

/ 

A rater assi gnment sheet i s prepared 1 1 sti ng the 
sequence of folders that each rater who has 
qualified will score. Each folder of essays will be 
scored by two raters. These raters are randomly 
assigned so that no two raters are systematically 
paired. j 

Sheets for recording the scores that raters \jive to 
training papers, qualifying papers, and actual test 
papers are prepared. 



Providing the Rater Training - 

Step 1: The training begins with a brief description of 
the scale's structure and rationale. Discussion 
deals with the broad features of the scale to be 
used, its research basis, and the need to develop 
common understanding of purpose and to maintain 
high levels of rater agreement. 

Copies of the expanded scale are passed out and 
read by the raters. 



)l 4 
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The trainer and the raters discuss each element 1n 
the scale being used. Discussion includes the 
rel atlonship between the seal e 1 s general 
competence rating and ratings on the subsequent 
subscales. Each of the subscales 1s defined and 
examples of the writing elements 1t focuses on are 
provided. 

Discussion then focuses on the six-point rating 
system appl 1 ed to each subscal e. Partlcul ar 
emphasis 1s placed on the operational criteria 
provided for each possible rating. The point of 
this discussion 1s to ensure common understanding 
of each subscale and what each possible rating 
means . Any 1 anguage cl ari f 1cat1 on necessary i s 
entered directly on the scale, which 1s then used 
for the remainder of the training. The trainer 
points out that levels of rater agreement which 
are lower than .80 (that 1s, where raters are 1n 
lower than 80% agreement) will suggest less .than- 
uniform scale application. \ 

The trainer passes out the prepared tralmVg 
packages of student writing samples. Raters are 



writing samples 1n their pacjeagei - The major 
Intention here 1s for raters ' to begin "practicing" 
the criteria on a few papers. 

Discussion begins after raters have scored the 
first three training papers. The trainer presents 
the scores that the experts gave to these papers 
and explains the reason for these scores by 
reading through the essay and noting the features 
that Ted to the score 1t received on each 
subscale. / 



d1 rected to read and then 
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Step 5: 



Subscale 



Raters rate another three 
displays the scores that 
using the following 



papers, and 
raters gave 
tabular format: 



the trainer 
each essay, 



Rating 



The trainer then presents the expert scores for 
these papers, and asks one or two Individuals who 
gave scores that are two or snore points away from 
the expert score to explain . their reasoning by 
Identifying features 1n the papers that exemplify 
a score category described in the scale. The 
trainer then explains the basis for the expert 
scores. 



Step 6: Depending on the level of/ agreement among raters 
after they have scored [the first six papers, 
discussion may follow a variety of \top1cs: 

' I . ■ \ 

° Is there need for further refinement of the > 
language of the subscales? \ c \ 

° is Everyone bringing a common framework to the 
use of the scales? do some raters j have, routine \ 
classroom diagnosis In mlnd/wht^e others are 
apply 1 ng the seal es from the ,standpo1 nt of 
minimum competency? ' j 

° do these differing frameworks suggest more 
lenient/more stringent application of criteria? 
1s that appropriate? ^ 

Step 7: Raters then continue to rate training papers 1n 
sets of five. After each set has been rated, the 
trainer distributes feedback sheets for two or 
three of the\ papers in the set. These papers are 
discussed only if questions; are raised. Group 
_ discussion focuses on the other papers in the set,- 
following the strategy described in Step 5, , and 
emphasizes those subscales where agreement levels 
are less than .80, that is, where less than 80% of 
the raters are in agreement. 
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After 10 to 15 papers have been scored, discussion 
and focus may become more directive. For example, 
1f only a few Individuals are giving discrepant 
scores, the trainer may work with these 
Individuals while they are scoring a set, 
I 

Step 8: When 80% agreement levels are reached on the third 
or so set of training papers, the trainer may 
decide that 1t 1s time to . distribute the 
qual 1fy1ng papers. This set contains 15 to 20 
papers which raters score without interruption or 
discussion. To qualify, raters must agree with 
each other within one point on each subscale on at-,, 
least 80% of the papers. 

If most of the group qualifies and 1s ready to 
proceed, raters then begin the Independent rating 
of the actual student test papers. 

At th si.-.ie time, additional training may continue 
on a subscale with less than 80% agreement. For 
example, Individuals who do not agree at the 80% 
level may continue to practice and discuss papers 
v individually or 1n small groups with the 
assistance of the trainer. 

Independent Rating of Essays 

Step 1: ' Raters refer to the rater assignment sheet to get 
the folders of essays they will score. After 
raters have scored one or two folders, they rate 
the set of three check papers and record the 
scores they gave them. They then turn 1n their 
check^scores, and read the feedback sheets for the 
three papers. The point here 1s to determine if 
the pair of raters scoring the same set of test 
papers are (1) in agreement with each other and 
(2) in agreement with the scores given by the 
"expert" rater. Since students will actually be 
graded 1n the process, It Is critical that raters 1 
scores be no more than one' point off the expert 
rater's score. Where greater differences exist, 
trainer and raters discuss the paper in question 
to resolve discrepancies. . 
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\ Step 2: Depending on the number, of actual student essays 
\ to be rated, independent rating may continue over 

\ - a few days, perhaps for a few hours each day. In 

x ; this case, at the beginning of each rating day, 

raters may begin by reading and rating a set of 
check papers. This serves to keep raters on track 
and tc minimize the re-emergence of idiosyncratic 
criteria. 

/' 

Summary 

Routine assessment of student's classroom progress may not require the 
careful check process described above. But it is another .matter when large 
numbers of raters are to judge the writing performance of large numbers of 
students, say in a minimum competency testing program. . Here the check 
procedure is critical, given the possibility of rater disagreement emerging 
over time. 

We strongly recommend, further, that when writing assessment results 
are to be used to make important decisions about student ability, each 
student's composition be read and scored, as described above, by two 
raters. If the two raters are in agreement at the .80 level or higher, the 
student's composition can be assigned the average cf the two scores. If 
the two raters show agreement lower than .80, then the student's 
composition should be rated again by a third, independent judge. * The two 
— raters— and— the-^Judge-ean-^ ther^-^euss^i^fferences^vi^the kind of process 
described in Step 5 citing features of the subscale element in the 
student's composition that match or exemplify the scale criteria for the. 
score given. This discussion should lead to resolution of the rating 
discrepancy. 
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CONCLUSION 

J 

In this paper we describe the need for using accurate and reliable 
criteria for the assessment of student writing. We offer scales for expo- 
sitory and narrative writing that we believe meet the standards of accurate 
and fair assessment. I 

Both our research in the assessment of writing and the technical 

assistance we have provided in several settings indicate ^that large numbers 

\ i 

of raters can be trained in the use of the scales, and that during straining 
and independent rating they can achieve and maintain high levels of 
agreement. Depending on the numbers of essays to be^^£d_jmd_the numbers 
of people who will be working as raters, the kinds of check procedures we 
describe are critical to the achievement and maintenance of high levels of 
i nter-rater rel iabil ity . \^ 

It may be, at least initially and until raters become familiar with 
the scale procedures, thct significant investments of time will need to be 
made. However, in the long rui^, if we are concerned about linking assess- 
ment with instruction so as to provide information for instructional 
improvement, the time invested can lead to high returns.' 

For example, in schools where £he CSE, scales are used in the writing 
assessment program, teachers state that the training and the scales have 
helped them to plan and monitor instruction. They have become more aware 
of specific writing skills and are directing their instruction to scale 
elements that result in improved student writing. 
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APPENDIX A 
The CSE Scales 
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Expository Scale 



Global Rating Procedures 

The purpose of global rating is to form a single judgement of how well a 
piece of writing communicates a whole message to the reader . Global 
scoring assumes that each characteristic that makes up an" essay ~ 
organlzation of ideas, content, mechanics, and so on ~ is related to all 
other characteristics. It further assumes that some qualities of an essay 
cannot easily be separated from each other. In shor^t, the procedure views 
a piece of writing as a total work, the whole of which is greater than the 
sum of its parts. 

Discerning readers naturally will attend to, or be influenced by, some 
essay characteristics more than others. In this general scoring, however, 
readers should arrive at a judgment regarding the essay's overall quality. 

For this element, you are being asked ° to . form an overall judgement 
concerning the effectiveness of the essays as examples of expository 
writing. The judgement ; should consider all the elements on the scale ~ 
essay organization, paragraph organization, support, and mechanics. 

Some views on exposition are given below: 

° Exposition is the kind of discourse that explains or clarifies 
^ a subject. 

° Exposition seeks to explain or inform through such methods as 
giving reasons or examples, comparing and contrasting, 
defining, enumerating, or through a combination of methods. 

° Exposition explains why or how. 

° Exposition promotes reader understanding of a subject. 

ELEMENT 1 

v.. 

General Competence 

Read each essay as a whole, first, 1n order to form an overall judgement of 
its quality.^ To assign the essay a score, consider the following 
questions: To what extent does the essay achieve an expository purpose for 
the intended audience? To what extent does the essay organize Its elements 
to create a effective whole? 
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Assign each paper a mark of 1-6 using the scale below: 

MASTER 



6 = Very competent 



5 = Definitely competent 



4 = Adequately competent 



The paper executes al 1 the el ements- 
competently. There are no serious errors. 
The paper has a clear main idea, logical 

organization, relevant, detail ed^support, 

and a command of basic mechanics. There are 
no major flaws. 

The paper 1s competent 1n all of the basic 
elements, but there may be a few minor flaws. 

The paper 1s adequately competent 1n all 
elements. There may be a few flaws. Some 
may be serious.* 



3 = Almost competent 



2 = Not very competent 



1 = Not at all competent 



NON-MASTER 

The paper lacks competence in one or two 
elements, and there are several major flaws. 

The paper has two or more of the elements. 
There are many serious flaws. 

Paper has none or only one of the elements 
competently executed. 



* If the essay 1s only one paragraph, paragraph cohesion 1s not considered 
a missing element, 1f the one existing paragraph coheres. If 1t clearly 
should have been divided Into several paragraphs, then paragraph cohesion 
1s a missing element. 
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ELEMENT 2 



Essay Organization and Coherence 

This subscale focuses on the flow of ideas throughout the entire paper and 
between paragraphs. The emphasis is on vertical relationships of ideas 
throughout the essay. 

i - 

The paper has a main idea (stated or clearly implied) which .makes a point 
about the subject and is at a greater level of general ity than the other 
points within the paper. Subtopics are logically related to the main idea 
and to each other. 



6 = 0 The subject is identified. 

0 The main idea is stated or implied in opening and/or closing- 
statement. 

0 Opening and closing statements match or logically relate to the i 

and to each other. 
° The topic is 1 imited 'through reference to key points or lines of 

reasoning* 
0 The essay plan is logical. 

0 The essay plan is consistently maintained (no digression or 
extraneous material). — 

5 * °~Tfiersubject is identified. 

0 The main idea is stated or implied in opening and/or closing 
statement. 

° Opening and ^closing statements relate to or follow from the text 
from each other. 

° The topic is partly limited by indicating number and type of key 

points. \ 
0 Plan is logical . 

0 Plan is signalled by appropriate transitions. 
0 There may be digression or an elaboration. 

• I ' • 

4 » ° The subject is identified. 

0 The main idea is identified or implied. 

0 There may or may not be an attempt to limit the ionic, give 

directions to subsequent reasoning, r .* \ 
° There may be a few minor digressions^rrom the plan, but no major 

digressions. ~ 
0 Subtopics can be reshuffled. 



MASTER 
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NON-MASTER 

Subject 1s clear. 

Main point may not be very clear . There may be a major digression 
or several minor digressions. 

A plan 1s attempted which may need to be inferred. 
Subject 1s clear. ~ Z 

Main Idea 1s not very clear and/or there may be more than one.^ 
The plan 1s attempted, but not consistently or not comp1ete1y v 
carried butv > . 
There are many digressions. 

Subject 1s ijnclear. 

Main Idea, Is absent or very unclear. 

No plan 1s attempted or followed. 



ELEMENT 3 



Organization - Paragraph* 

This subscale focuses on the relationship of ideas within paragraphs; their 
logical interrelationship and subordination to the paragraph topic. 



Paragraphs present subtopics which are developed by cohesive groups of 
supporting * statements* Each subtoplc represents a complete unit of 
thought. Major units of thought are delineated by physical separation of 
paragraphs. Statements within the paragraph relate logically to each other 
and to the paragraph subtopic. 

MASTER 

6 = ° All major units of thought are set off by distinct paragraphs. 

° The paragraph has a clearly stated or Implied topic. 

° All sentences within paragraphs are related to each other and to 
the paragraph topic, and are subordinate to it. There are no 
digressions or Irrelevandes. There are no one-sentence paragraphs 
unless they are especially effective. 

5 = ° Most major subtopics are developed 1n paragraphs. 

° fflosf paragraphs contain logically related subordinate support. 
° There may be a minor digression. 

4 = ° Many subtopics are developed 1n discrete paragraphs with related 
subordinate support. 
° There may be some minor digressions. 



NON -MASTER 

3 - ° In some paragraphs statements are logically related, but may not 
function as support subordinate to the paragraph topic. Paragraph 
separation 1s evident and consistent. v Some relationships between 
sentences must be Inferred. 

2 = ° There are few paragraphs where statements are logically related or 
supported. There are many digressions. Paragraph separation 1s 
Inconsistent. Many relationships among sentences must be inferred. 

1 = ° There are no paragraphs v.here statements logically cohere. 
Paragraph separation 1s incorrect. 



* For conventions for paragraph separation (e.g. , physical separation or 
Indenting) see Mechanics. 
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ELEMENT 4 



Support 

This subscale focuses on the quality (specificity and relationship) of\ the 

support provided vertically for the essay theme as well as horizontally 

..wjJtMllJLa^p \ „ ■ f . 



Support statements are at a greater level of specificity than the general 1« 



ate 



zations they are Intended to support. Support statements logically rel 
to each other and to the generalization. Support includes specific details 
such as examples, facts, anecdotes, reasons, and concrete language. 



MASTER 



6 = ° The essay's main idea and all paragraph topics are supported by 
relevant, specific statements. ■ 

5 » d The essay's main Idea and almost all paragraph generalizations/ 
assertions are supported by predominantly specific statements. 
Enumerations are supported by descriptive detail, functions, or 
rationale. , 

4 * ° The essay's main idea and most paragraph generalizations are 
supported. Most support is specific. Enumerations are supported by 
descriptive detail, functions, or reasons. 



NON-MASTER 

3= ° Some or all generalizations are supported by logically rel ated. . 
detail , or some support -is not specific but 1t is distinct and 
clear. Support may be primarily an unelaborated, undetailed, 
unsupported list. 

2 = ° An attempt 1s made to support generalizations/assertions. Some 
supporting sentences do not logically follow from each other or are 
redundant. 

° Support lacks precision, clarity 1n details, and/or language. 

1 =. ° No support is provided, or, 

° Support, if present, is vague, and confusing, or, 
° Not logically related to generalizations, or, 

° At the same level of generality as the topic it attempts to support. 



34 



- 31 - 



ELEMENT 5 

Grammar/Mechanics (Usage, sentence construction, spelling, punctuation, 
capitalization) ; ' ' 

Errors in grammar or mechanics are considered according to how/seriously 

„they~iJiteEf.eLrs^ Ib£3e_«XDJ3^ 

global error makes it difficult to understand the writer's message. The 
sentence, "Tomorrow, I went to the store," for example, forces the reader 
to decipher which time context (future or present) the writer is actually 
alluding to. A local error does not seriously interfere with the writer's 
message. For example, in the sentence, "He/ going to the store now," the 
message is clear but the grammar is incorrect. Naturally, an overabundance 
of errors which, if individually considered are local, can seriously 
distract the reader's attention and understanding. The intent here is to 
evaluate errors in relation to how much they interfere with the writer's 
effectiveness in communicating rather than/ to attempt to assign different 
values to the myriad of possible grammatical and mechanical errors that can 
occur. 

MASTER / 

6 = ° There are few or no errors. There are no serious errors. 

/ • /' ' 

5 = ° There may be a few minor errors ir] the- categories, but no more than 
one serious error. 

4 = ° There are some errors. A few may /be serious. 



NON-MASTER 



3 = ° There are numerous errors in the categories. There are some 

serious errors in several categories. Below mastery in sentence 
construction. / 

2 = ° There are many serious errors, ^causing some confusion. 

1 = ° Errors are so numerous and ser/ious that they interfere with 
communication. 



\ 

> 
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Check those mechanical skills below master level. 

Usage. Does not display command of standard vocabulary usage. 

Sentence construction. Does not display command of basic sentence 

structure. 
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Spelling. Misspells many common words (includes homonyms! . 

Capitalizations and punctuation. Does not use standard conventions 
appropriately, e-g., periods, commas, capitals, apostrophes. 

Conventions of paragraph separation are incorrect or inconsistent 
(indenting, spacing, titles, numbers). 



1) Usage 

Serious errors: 

> 

- Homonyms, e.g., it, it's; their, there; to, two, too 

- Incorrect use of common words 

- Incorrect pronoun reference 

Minor errors: 

- awkward or odd use of words, phrases, butmeaning still clear 

- vague, abstract language 

2) Sentence construction 
Serious Errors: 

- Subject verb agreement 

- Run on 

- Fragments 



3) Spelling, 



SeriousX^rrors: 

- Common Vords misspelled; does not include homonyms. Any misspelled 
word only^counts as one error, even if the misspelling repeats. 

Minor .errors :\^ - 

- Unusual, le^S\frequent words 

\ . 
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4) Capi tal i zati on/Punctuati on 

- Initial capitals — common proper nouns 

- Periods at end of sentence, common abbreviations 

- Commas (in series, for "opening phrases) 

- Contractions 



5) Paragraph Conventions 
Serious errors: 

- Title 

- Number . / 

- -Inconsistency of separation convention 

- Absence of any convention for separation 

Minor errors: 

- Use of spacing instead of indentation (as in business letters) 
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Narrative Scale 



Global Rating Procedures 

The purpose of global rating is to form an overall judgement of how well a 

.plece^gf-jwritlog^ to the reader. Global scoring^ 

assumes that each element of an essay is integrally related to other 
elements in the essay and the effect is cumulative. In short, the 
procedure views a piece^ of writing as a total work, the whole being greater 
than the sum of its partes. 

For the element of "General Competence," you are being asked to form an 
overall judgement concerning a composition's effectiveness as an example of 
narrative writing. All \of the elements presented in the scale— focus/ 
organization, support, and grammar/mechanics— should" be considered in 
rating for general competence as well as any other elements which you may 



feel are important. 

> ELEMENT 1 

General Impression I 

Read each essay as ai whole, first, to form an overall judgement of Its 
quality. To assign the essay a score, consider the following questions: 
To what extent Is the essay an example of effective narration? To what 
extent does the essay or/ganize its elements to create an effective whole? 

MASTER 

An excellent example of narration. Each element 1s evident in use 
throughout the essay (e.g.* topic Is clearly Identified, characters 
defined,, situations fully developed). 

A good example narration, but the elements are not equally well 
developed throughout the essay/paragraph. 



4 - An adequate exampl 
ments of narratfor 
presents nothing 



e of narration. The writer Incorporates the ele- 
The essay is simple, Informing and clear, and 



presents nothing more than essentials, mere may oe one or two iso- 
lated Instances of (global error and no wjore than three local errors or 
a total of five /local errors. / 



re than essentials. There may be one or two iso- 
gl« 
cal 

NON-MASTKR 

3 * A marginal example. The writer presents evidence of limited skill in 
using the narrative elements. The elements are all present but 
developed poorly. Extremes are noted: strong to weak) subject focus, 
proper to poor/ grammar and mechanics. ^ 

2 a A poor exampl/e. Writer's use of elements 1s problematic, focus on 
topic steadily decreases, support statements, 1f present, are weak; 
grammatical and mechanical errors also present. 



1 = This paper 1s either off topic or genre. Presents no central subject, 
supports are irrelevant, or absent, and contains numerous grammatical 
and mechanical errors. The essay cannot be comprehended. 



ELEMENT 2 



Focus/Organi zati on 

This subscale examines whether the topic Is clearly indicated and developed 
iti-an-organized manner. The composition-should exhibit a clear structure 
both within and between paragraphs. The topic should be limited and free 
of extraneous material. 

. ; MASTER 

6 = 0 The subject of the composition Is clearly stated or Implied. 
0 The topic 1s clearly limited; there are nb digressions or 

extraneous material. 
0 Events and/or Ideas &?e presented 1n a logical manner; they are 

linked both within and between paragraphs. 
° The subject 1s developed through description of events, setting, 

or through the thoughts, emotions of the characters Involved., 

5 = ° The subject of the composition 1s clearly stated or Implied. 

0 There may be one of two brief digressions v or elaborations, but the 

topic 1s clearly limited. 
0 Events and/or Ideas are presented 1n a logical manner; they are 

smoothly linked both within and between paragraphs. 
0 The subject 1s developed through description of events, setting, or 

through the thoughts and/or emotions of the characters Involved. 

4 = 0 The subject of the composition 1s clearly stated or Implied. 

0 There may be one or two brief digressions or elaborations, but the 

topic 1s clearly limited. 
° Events and/or ideas are presented in a logical manner, but Unking 

within and/or between paragraphs may be weak 1n one or two 

instances. 

0 The subject 1s developed through description prevents, setting, or 
through the thoughts and/or emotions of the characters Involved. 

0 The writer gives evidence of knowing the concepts of focus and 
organization. , 



NON-MASTER _ 

3 -a 0 The subject of the composition 1s stated or Implied. 

0 The logic or progression of Ideas/events within or between 

paragraphs 1s often unclear; Unking 1s frequently weak. 
0 The topic development through events, setting, or character 

thoughts/feelings 1s uneven; there may be too much or too — 

little elaboration of some aspects. 
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The subject of the composition is stated or implied, but the main 
point is not clear. 

There are many digressions or elaborations; the topic is not 
limited. ' , . 

The logic or progression of ideas and/or events is so unclear that 

no story line is discernible. t 
There is little development through events, setting, or characters 
feelings or thoughts. 

The subject of the composition is unclear or absent. 
There is no obvious organizational plan. 
There is no development" of events, setting, or characters. 
There is much extraneous material. 



ELEMENT 3 

o 

Support 

This subscale focuses on the quality (specificity and relationship) of the 
support provided for the essay theme both within each paragraph as well as 
throughout the essay as a whole. 

Support statements should be at a greater level of specificity and depth 
than the generalizations they are Intended to develop. Events, descrip- 
tions, and characters, should be developed through the use of specific, 
well -integrated tietaliY, such as examples, facts, anecdotes, or descrip- 
tions. These details should provide the reader with an image of the 
appearance, feelings, thoughts, actions, or mood of the events taking place 
in the narrative. ^ 

MASTER / 

6 = 0 Events, characters, And/or descriptions are' developed by specific 
and clear supporting details, such as examples, descriptions, 

anecdotes, facts, etc. \ v 

0 Supporting details provide an image/feeling of actions, appearance, 

feelings, thoughts, and/or mood 1n the paragraph/essay. 
0 Supporting details are more specific than the general ideas/events; 

characters are described and well integrated to the rest of the 

paragraph/essay. 

/..* ; . * •* 

5= 6 Events, characters, and/or descriptions are developed- by specific 
and ' clear supporting details, such as examples, descriptions, 
anecdotes, facts, etc. 0 * 

0 The use of supporting detail Is not consistent throughout. For 
example, J h one or two instances, the writer may not [provide Infor- 
mation about appearance, feelings, thoughts, actions, \or mood. 
0 Although most supporting statements or details provide , 1n-depth 
descriptions and are more specific than the general statements they 
describe, there may be one or two instances where the detail lacks 
depth. * ./ -I 

4 = 0 Except for one or two instances, events, characters, and actions are 
descri bed through the use of adjectives, adverbs, prepositional 

pfirases7~~Stcr- 

0 Events, characters, and/or actions are devel oped . by supporting 
detall^r 

0 The use of supporting detail may be inconsistent or rudimentary. 
One aspect mentioned in the essay (I.e., character, event, or 
description) may not be sufficiently developed. Overall , however, 
the writer gives evidence of using supporting detail to develop most 
aspects of the essay /paragraph. 
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3 = 



2 = 



NON-MASTER 

0 The use of detail 1s very uneven. Several statements/descriptions 
are not developed through the use of detail. 



0 There Is too much superficial or Irrelevant detail. 
0 Some of the detail 1s not well Integrated 



within the 



paragraph/essay. 



b 



There are 
to develop 



very few 
events, 



Instances where 
characters, or 



There 1s too little detail, 
supporting detail is used 
descriptions. 
Most details lack specificity and depth. 
The details are not smoothly integrated in the composition. 

Supporting details are vague, confusing, or not related to the 
events, characters, or descriptions they are meant to describe. 
There is little or no evidence of supporting details in the 
paragraph/essay. 



\ 



\ 
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ELEMENT 4 



Grammar/Mechanics 

Errors 1n grammar or mechanics are considered according to how seriously 
they Interfere with communication. These errors may be global or local. A 
global error) makes 1t difficult to understand the writer's message. The 
sentence, "Tomorrow, I went to the store," for example, forces the reader 
to decipher which time context (future or present) the writer 1s actually 
alluding to. | A local error does not seriously Interfere with the writer's 
message. For example, 1n the sentence, "He going to the store now," the 
message 1s cljear but the grammar 1s Incorrect. Naturally, an overabundance 
of errors which, 1f Individually considered are local, can seriously 
distract the reader's attention and understanding. The Intent here "is to 
evaluate errors 1n relation to how much they Interfere with the writer's 
effectiveness 1n communicating rather than to attempt to assign different 
values for the nyrlad of possible grammatical and mechanical errors that 
can occur. 

MASTER 

6 s ° The writer appears to have control of the usage and mechanical 
aspects of this essay. 
° There are no global errors . Recall that global errors affect the . 
essay/paragraph as a whole and Interfere with clarity of 
communication. 

° There are only one or two local errors (if any) in mechanics. For 
example, there may be a few spelling errors of difficult words, 
e.g. antenna. 



o 



o 



Usage and mechanics are not a problem 1n this paper. 

There are only a few local errors 1n usage or mechanics. 

There may be one- Isolated 9lobaT error , but the general meaning 1s 

clear throughout the paragraph/essay. 

Usage and mechanics are not a problem 1n this paper. 

^Errors do not Interfere with the clarity of communication. For 

example, confusion of to, too, two; their, there; or other local 

errors. 

There may be one or two isolated instances of global errors, and no 
more than three local errors, or a total .of five local errors. 



NON-MASTER 

Some errors, global and local, do interfere with the clarity of 
communication. For example, there may be a long run-on sentence, 
Inappropriate fragments, or Incorrect tense continuity (e.g., 
Yesterday we are going to school.") 

* is 
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2 - 0 Errors may detract from the clarity of communication, such as 
confusing antecedents, omission of .key words, serious misspellings 
of common words (e.g., confurmable/comfortable, laike/lilce). 
° There are many-global and local errors throughout the 
essay/paragraph. " 

1 = ° Errors make this paper very difficult to read and understand. There 
is an overabundance of global errors and a significant amount of 
communication is lost. 
° Errors are not restricted to one type of problem, such as run-on 
sentences. 
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. Mechanics/Grammar 
Examples of Errors 

I. Senl/enc^JIonstructton 
Global errors 

- tense continuity (e.g., Yesterday I go to ...) 

- very long run-on sentences 

- fragments 

Local errors 

- subject/verb agreement (He go to the store.) 
-. short run-ons 

- incorrect or lack of connectors between clauses 

II. Usage 

Global errors / 

- Incorrect use of common words 

- code switching (mixing languages) (Today I played with my sister y 
fu.1mos to the store.) ^ 

Local errors 

- homonyms (e.g., it, It's; their, there; to, two, too) 

- Incorrect pronoun reference (That 1s she book.) 

- awkward or odd use of words, phrases, but meaning still clear 

III. Spelling 
Global errors 

- Common words frequently misspelled; does not Include homonyms. Any 
misspelled word only counts as one error , even 1f the misspelling 
repeats (comfortabl e-confurbal ; should not-shurent; wrong-rong) 

- Words misspelled so seriously as to Impede communication (e.g., mild 
down/melted down) 

Local errors 

- unusual, less frequent words 
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IV. Punctuation/Capitalization 
Global errors 

- periods at the end of sentences 

- contractions 

- commas where understanding 1s Impeded 
Local errors 

- Initial caps —common proper nouns 

- commas (In series, for opening phrases) 

- periods fpr abbreviation 
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• APPENDIX B 

Student Compositions Rated 
During a Training Session 
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Sa mples of Student Compositions Rated 
During a CSE Training Session 

-The Jfiv_e_studentjwrj.tln rated during a 



CSE rater training session for secondary school teachers (Quellmalz, 
1982). The samples are representative lOth-grade student exposition, and 
were among those used early in the training as a check on how uniformly and 
consistently raters were applying the criteria. 

The rated writing samples show the level of agreement between two 
rater trainees. They also show how closely each seemed to be following the 
criteria agreed upon, rather than applying perhaps more idiosyncratic 
values. 

At the top of each student composition are two numbers. These numbers 
show the General Competence rating that the two raters \ independently 
assigned to the sample. These general impression ratings indicate each 
rater's assessment of how well the student's writing met the specific 
skills required in Focus/Organization, Support, and Grammar/Mechanics. 

Each sample composition is followed by a feedback sheet showing a CSE 



rater's judgment of its general competency level, as well as of now well it 
dealt with the skills addressed in the three remaining subscalesA / 
Asis readily seen, there are high levels of agreement between the two 



raters, and similarly acceptable agreement between the raters land the 

< \ 
assessments provided on the feedback sheets* Had these agreement levels 



not been achieved, the feedback sheets would have been used as 



a focus for 



discussing the causes of rater disagreement and for resolving differences^ 

,/ - . \. 

before taking up the next set of compositions to be rated. \ 
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Set Number Training Check ! 
Code Number ' 



C2) 



FEEDBACK SHEET 



ELEMENT 


f 

RATING : V . • 


General Competency 


The main point is clear and specifically supported. 
4 Some mechanical errors and awkward usage. 


• • \ 
Focus/Organization 


Paper has main idea summarized in the opening 
and closing paragraphs. Ho use of transitions. 


I 

Support 

• 


/ 

./ ■ ■ . • •. 

4 Although there is only one example* it is clear 
and specific. * 


Grammar/Mechanics 


>■■" \. . ■;' -:'/. 

4 There are a few minor errors. n If a person... they." . 
"Brakie." 

• ! ' IS 
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Mechanical Skills below mastery 

Sentence Cons tructi on 
' Usage : 

1 Spelling 

Punctuation and Capitalization 
■ ■ -■„■., :•' Paragraph Conventions 
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..l_.".!Qth GradL. Prompt. — — : •— ~ 

!. x3rat/& cl. prcmibe.?-/rs.- procci\s^Jjo-56r?)8jz-- 

'. Hrh i nq/cSoflL .exavje and Qea&upr&d . 5o.hLJt£0&~* . 

it \9> .important to teepj t~. £C..:sj uu — — : — 

Slii tVcm^^. e&xxjld niv/er.be- toroVt*LO,..I£ : r 

vjou .^romi'be. sctfr^tfcnnq /NOuu sJoou-ld 

cb(nft--tbinK. \jou cQr\t .k€€p .a. pronQi^:>joa— 

- for-.a-datc cnfi-. davjj ^ho^.i^^Pr.iend^-- — 
&att^„ioa€> over, ^ 

ge^r JrW: makeup Prono . he^r pur <bLj 

cl pacK\ d? p\\te fell a^t.. ..oh&n . P&tfy . T „ 

ae>H&d . Wn u±r\t th^ .coere v <oalL/ 

-fold ..^^ ocuid YuLf^<^^~ 
pra^i^^ ^n0t io ^si a^M^^^a to a^v%n3 v 




„. f^alkj em(dfe.ou^ x dc> vjc>u."Mnin\<: tht "LaJI/irLftt .... J... 
: ._./...fc^.-OnvjrnDr^? Q3=4^_--furdvj^ -U — 

h&c^ct&^JioJ}^ 



:.Pcomfiif2S_dxu fcirft. \^.^\d^i(i-^qJ- doat T — — •_ 

^ee\..>{cUXaQJsiL^pJ^* ,9©p&..iJ^^ __ 

_„...tx.Djv\vi^_^>\l\ .rot b^Jrcur&3dj£^^ — 

.... ufon u^ib . re^C^X^ :.— : — 

. r . ...be- xze&cteti.^ • -— : 

.J. J; S]ou pronMb£,,- . .-^ . _. . — '—— — - — 
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Set Nuinber Training Check 6 
Code Number 



C7) 



FEEDBACK SHEET 



/../.. i'\ 

/ ELEMENT 

/ i.U.i.1 il.ll 1 


N 

RATING 


/ General Competency 


Paper is generally well developed, but flawed by 
3 <• 

poor paragraphing and numerous mechanical errors. 


V 

Focus/Organization . 


Position is clear and logically developed, but 
3 paragraphing is a problem. 

Good conclusion. • 


/ 

Support 


5 Gives a specific example to support thesis. 


• . ■■ y 

Grammar/Aecharii cs 

M ^7 •■ ■: 


• ; ' "3 ' ' , * * ■ . — 
3 ^Numerous awkward usages. 

Problem v/ith use of commas. 

. / ■ ■ ■ . ....... 



',/ J . Mechanical Skills below mastery 

'Sentence Construction 



y Usage 
j Spelling 



y_ Punctuation And^Capital izati on 



Paragraph Conventions 
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x Set Number Training Checks? 
Code Number 



FEEDBACK SHEET 



" . / 
ELEMENT 

/' 


RATING 

' / 




General Competency - 


The line of reasoning is confusing. Support is . 

2 

vague and mechanics are intrusive. 


* . \ 

\ 

; Focus/Organization 


The main idea is unclear and there is no conclusion- : 
L. raper moves Trout uenmuion ot pronnses to 
"very often they should be broken. 11 . 
No evidence of paragraphing. 


... 

' Support 


2 Support is attempted, but not specific. 

v ' : ■ % 


Grapnar/Mechani cs 


V ■ ( ■ ' ■ ■ ■ ,: : 

Sentence errors. • , : 
3 Some awkward usage.. "We use promises. . .* : 
* "Like when..." "Get caught in the middle of two 
people." [ V ' 



Mechanical Skills below mastery 

/ Sentence Construction 
■» — — 

Usage 

Spelling 

Punctuation and Capitalization 
. Paragraph Conventions 
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Set Number Training Check 4 
Code Number 



FEEDBACK SHEET 



— "T 

ELEMENT 


RATI^ 


General Competency 


Main point confusing because of poor logic and 
2 numerous mechanical errors. 


I Focus/Organizati on 


2 The logic is unclear. The paper confuses promises with 
lying. 


*■• 

j . j 

Support 


2 Support is attempted* but unclear. 

• 

• 


Grammar/Mechani cs 

i. 


Major usage, sentence construction and spelling 
1 errors interfere with meaning. . 



Mechanical Skills below mastery 

t/ Sentence Construction 

\fvjk ' S Usage 

I"* 8 " . %/ Spelling 

Punctuation* and Capitalization 

* Paragraph Conventions 57 
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Set Number Training Check 2 
Code Number ___ 



(3) 



f h 1 

ELEMENT 


RATING 


General Competency 


3 The papers 1 point is at times confused by 
imprecise language. 


Focus/Organization 


Logical development unclear. 

2 

"promise = blackmail" 

"promise = are big and small things" 


Support 

* 


2 Support is not very specific or clear. 


Grammar/Mechanics 


3 Usage is a pervasive problem. "For now on" "Like ; 
she wanted me to. . ..slang terms. . , • ; v. > 
Subject-verb agreement. 



« 



Mechanical Skills below mastery 

, Sentence Construction 

y/ Usage 
_. Spelling 

- ,., Punctuation and Capitalization 
Paragraph Conventions 
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